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Abstract 

Designing protocols and formulating convenient programming units of abstraction for 
sensor networks is challenging due to communication errors and platform constraints. This 
paper investigates properties and implementation reliability for a local read-write abstrac- 
tion. Local read-write is inspired by the class of read-modify-write operations defined for 
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c/2 , shared-memory multiprocessor architectures. The class of read-modify-write operations is 



important in solving consensus and related synchronization problems for concurrency con- 
trol. Local read-write is shown to be an atomic abstraction for synchronizing neighborhood 
states in sensor networks. The paper compares local read-write to similar lightweight op- 
erations in wireless sensor networks, such as read-all, write-all, and a transaction-based 
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\q ' abstraction: for some optimistic scenarios, local read-write is a more efficient neighborhood 

. operation. A partial implementation is described, which shows that three outcomes charac- 

terize operation response: success, failure, and cancel. A failure response indicates possible 
• inconsistency for the operation result, which is the result of a timeout event at the oper- 

ation's initiator. The paper presents experimental results on operation performance with 
different timeout values and situations of no contention, with some tests also on various 
neighborhood sizes. 



1 Introduction tations of (perhaps less orthodox) techniques 

from areas of signal processing, database, and 



X 

Wireless Sensor Network (WSN) platforms parallel or distributed computing. This pa- 
add a twist to traditional programming as- per follows the same research direction, ex- 
sumptions. Many resources can be quite ploring the adaptation of a read-modify-write 
constrained, including bandwidth, program abstraction as a unit of sensor network pro- 
memory, and platform computing power. Not grams; we propose an operation called local 
surprisingly, research on sensor network pro- read-write (LRW) for neighborhood commu- 
gramming to date has sought abstractions nication in a sensor network, 
and tools that can satisfy the resource con- 
straints, yet enable productivity in software The compare-and-swap (c&s) instruction, 
development cycles. Typically, these abstrac- available on many multiprocessor architec- 
tions are not entirely new ideas, but adap- tures, is an example of read-modify-write. In 
one atomic step, a processor executing C&S 
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word with the content of a register; the con- 
dition for this swap is that the content of the 
memory word have a prescribed value given 
as a field of the instruction or given in an- 
other register. This idea, that a single in- 
struction specifies a condition, a write value, 
and expects a response value, can be gener- 
alized and translated to the setting of nodes 
and packet-based communication. A simple 
instance of an LRW operation is illustrated 
in Figure [TJ Sensor node x initiates the oper- 
ation by transmitting a packet to neighboring 
node y. Node y inspects the packet, and pos- 
sibly schedules a tentative write to some local 
variables; then y transmits a response packet 
to x. Upon receipt of y's response, node x will 
either decide to confirm the operation or back 
out and void the operation. Voiding the oper- 
ation will result in y discarding its scheduled, 
tentative write. 
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Fi gure 1: LRW with one neighbor. 



Node z shown in Figure Q] lies outside x's 
neighborhood; there is the possibility that z 
could initiate an LRW operation concurrently 
with x, so that y first receives a packet from 
x, then a packet from z, and these requests 
conflict because they write to the same loca- 
tion. For these LRW operations to be atomic, 
the net effect of running both should be log- 
ically serial, that is, as though one operation 
completes before the other begins. The de- 
sign choice for this paper is that y should re- 
ject z's request while x's operation is pending, 
that is, y should immediately send a negative 
response to z. 



The single-node neighborhood of x, in Fig- 
ure [U can be generalized to larger neigh- 
borhoods, illustrated by Figure [2j Node x's 
LRW operates on a neighborhood of nodes 
yi through y^. Although the figure suggests 
k messages would be transmitted by 
single local broadcast suffices for many ra- 
dio platforms. A typical WSN application 
for LRW is data aggregation. Suppose each 
yi has recorded some sensor value di, and 
node x's task is to compute some function of 
{ di 1 1 < i < k} and save the result to its 
flash memory. After x has completed this ag- 
gregation, each yi can discard its di value and 
recycle local memory. Note that if yi were to 
asynchronously send di to x, it could be that 
x does not have local buffers available for this 
data; putting x in control is a way to man- 
age resources safely. In one LRW operation, 
x can collect all di values and also schedule 
the di variables at each yi for recycling. How- 
ever, if x does not collect enough di values, 
say fewer than k/2 neighbors respond to the 
request initiated within the LRW operation, 
then x could cancel the operation and retry it 
later. Classical applications of read-modify- 
write, such as consensus or leader election 
(applied to a WSN neighborhood) can easily 
be expressed as an LRW operation. 



Figure 2: LRW with k neighbors. 

Contributions and Organization. Sec- 
tion [2] summarizes related work. Section [3] 
specifies LRW properties and exposes some 
design choices for implementation. Section |4] 
presents a theoretical result showing how a 
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model based on this abstraction differs from 
other choices. Section \5\ contains implemen- 
tation results, which feature experiments to 
show design tradeoffs. Discussion of conclu- 
sions is in Section [6j 

2 Motivation and Related 
Work 

Several tracks of WSN research draw analo- 
gies to database and parallel computing meth- 
ods. Early proposals for querying sensor 
networks motivated protocols for aggregation 
and routing to support query language op- 
erations [ini [9]. The idea of program- 
ming a WSN whole (called macropro- 
gramming) sometimes take the position that 
programming sensors resembles the ensem- 
ble programming of parallel computing ma- 
chines, using SIMD or MIMD instruction se- 
quences pTJ [T2] . Inspired by distributed 
computing research, there are proposals to 
adapt such paradigms as snapshots, leader 
election, and wave computations in WSN sys- 
tems [131 Q21 [T3] . This paper draws analogy 
to instructions for atomic communication in 
multiprocessor, shared memory systems. 

In contrast to high-level concepts for WSN 
software, there is also significant research 
adapting the techniques of ad hoc networks, 
peer-to-peer, and even internet protocols to 
the needs of WSN applications and the lim- 
itations of WSN platforms. Two priorities 
for such research are reliable communication 
and power conservation. A question emerg- 
ing from this research is: what kind of com- 
munication abstractions will be convenient 
for programming (i.e., the interfaces are sim- 
ple and hide low-level complexity and prob- 
lems of heterogeneous platforms) while en- 
abling efficient use of resource? This question 
has predominantly been investigated with re- 
spect to non-local communication, for in- 



stance, multi-hop protocols, routing struc- 
tures, and middleware services for publish- 
subscribe abstractions. Our work looks at 
local communication, where "local" refers to 
single-hop communication, also called neigh- 
borhood communication. 

On one hand, the literature of MAC pro- 
tocols, specialized to WSN platforms, exten- 
sively explores the concerns of local commu- 
nication [3]. Platform hardware may directly 
support unicast and neighborhood broadcast 
operations, and some radio chips provide low- 
level support for unicast packet acknowledg- 
ment in one programmable operation. On the 
other hand, there are several papers [H [31 [2] 
suggesting higher-level programming units for 
local communication. A natural abstraction 
for local communication is atomic read-all, 
which is the operation of reading the local 
states of all nodes in a neighborhood. Us- 
ing atomic read-all operations, programs el- 
egantly express calculation of neighborhood 
statistics; Section [31 elaborates on a variant 
of read-all with stronger atomicity properties. 
Unfortunately, the read-all abstraction does 
not efficiently map to WSN platform abilities. 
An alternative abstraction is atomic write- 
all, which may be implemented by a single 
local message broadcast. A write-all opera- 
tion writes (some part of) the states of every 
other node in a neighborhood. This operation 
is not so natural for programming as read- 
all, however program transforms have been 
proposed that convert many programs using 
read-all operations into ones that employ only 
write-all operations [HI El [3]. Reliability is a 
concern with naive implementation of write- 
all consisting of a single message broadcast; 
the broadcast can lose messages to a subset of 
neighbors due to noise or collision with other 
message traffic, say originating from other 
neighborhoods in the WSN (in [4], the basic 
operation is called "write-all with collision"). 

The concerns of reliability and atomicity 
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are fundamental to database transaction the- 
ory, where ACID properties define correct 
transaction processing. The paper [2] sug- 
gests a local WSN operation motivated by 
database transactions: one atomic opera- 
tion reads from a subset of neighbors and 
writes to a subset of neighbors. To im- 
prove reliability, the local transaction imple- 
mentation consists of a sequence of messages: 
read-request, response, then write-commit 
or abort-transaction. Transactions may be 
aborted because of interference with contend- 
ing transactions, and the aborted transac- 
tions need to be retried. The reliability of 
such local transactions is imperfect: final 
commit messages can be lost and the transac- 
tion initiator can crash. Standard techniques 
that add reliability to database transaction 
processing, such as stable storage and trans- 
action journaling, are unrealistic for many 
WSN platforms. 

To give some idea of the resources needed 
for the operations discussed above, Table [T] 
summarizes optimistic, best-case resource 
measures for a neighborhood of n nodes. The 
two measures are number of messages (in- 
cluding both unicast and local broadcast mes- 
sages) and number of rounds, where a round is 
a time interval of sufficient length to allow all 
nodes in a neighborhood to send a message. 
The latter measure would allow for queuing, 
processing, and transmission delays as well as 
extra delays due to the medium access control 
layer for collision avoidance. The first row of 
the table reflects that a read-all operation is 
initiated by one node, followed by each of its 
n — 1 neighbors sending a response. A write- 
all operation potentially has the least resource 
cost of any operation, consisting of just one 
broadcast message; however to provide for re- 
liability, an implementation of write-all may 
return acknowledgments from each recipient 
of the broadcast back to operation's initiator. 
For this reason, the number of message prim- 



Operation 


Messages 


Rounds 


read- all 


n 


1 


write- all 


1 or n 


1 


transact 


2 + r + w 


2 


LRW 


n 


1 



Table 1: Operation comparison. 



itives is reported as "1 or n" in Table [TJ A 
local transaction, as defined in [2] and called 
"transact" in Tabled! has a read set of r nodes 
and a write set of w nodes. The transaction 
is initiated with a broadcast, followed by a re- 
sponse from each node in the read set. Then 
the transaction initiator transmits a broad- 
cast to the write set containing values to be 
written, and each member of the write set uni- 
casts an acknowledgment to the initiator; the 
acknowledgments are needed so that the ini- 
tiator can decide whether to allow the trans- 
action to commit or to broadcast a cancel 
message. The read and write sets may over- 
lap, with the worst case being r = w = n — 1 
(which would put the message cost of trans- 
act at 2n). The LRW operation begins with a 
broadcast, followed by each of the n— 1 other 
nodes responding. Since any value to be writ- 
ten is contained in the initial broadcast and 
responses are collected by the LRW initiator, 
no additional round is needed to complete the 
operation. 

The measures of Table Q] are optimistic 
numbers in two senses. First, the measures 
are for transactions that succeed, that is, 
they do not fail due to conflicts with con- 
current transactions or negative responses (an 
LRW operation would need to include a can- 
cellation message if any neighbor response 
indicated some unanticipated value). Sec- 
ondly, the table does not include commit 
messages for transaction or LRW operations. 
This is because sensor node timing and clock- 
ing mechanisms enable commit to be time- 
triggered, that is, each node commits a trans- 
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action after sufficient time has passed without 
receiving a cancellation message. 

3 LRW Design Issues 

Local Read-Write (LRW) is an operation de- 
fined on variables of WSN nodes. We assume 
that each node has the same set of variable^ 
that can be read and written by an LRW op- 
eration. For variable v and node q, let v q refer 
to q 7 s instance of v. Each invocation of LRW 
specifies: (i) a function / defined on a sub- 
set of node variables, (ii) a subset of node 
variables to be written, and (in) a boolean 
function g. Function / can be computed at 
any node, and either returns a negative re- 
sponse value _L or returns a pair (r, B), where 
r is a value provided for computing g and B 
is a list of values to be written to the vari- 
ables specified in (ii). A nonlossy LRW oper- 
ation is defined with respect to an initiating 
node p and p's neighborhood N(p), consist- 
ing of three steps: (1) for each node q G N(p), 
function / is computed; (2) function g is com- 
puted on the set of r-values { r q \ q G N(p) }; 
and (3) if the result of g is true, then B q is 
written to the write variables of q, for each 
node q. A lossy LRW operation would allow, 
in (l)-(3), proper subsets of N(p) to model 
the loss of messages. We do not formally spec- 
ify lossy LRW instances in this paper. 

Viewed from the application perspective, 
an LRW operation begins when initiating 
node p invokes LRW and ends when p receives 
a response from the LRW. Between the invo- 
cation and response, we assume that p does 
not invoke another LRW instance. Thus the 
only source of concurrency in the system is 
contention among LRW operations of differ- 
ent LRW initiators. The behavior of a set of 
(possibly concurrent) LRW operations can be 

"This assumption is not essential, but simplifies the 
description. 



specified by a sequence, called an LRW his- 
tory, which contains LRW invocations, con- 
tains results of / and g evaluations, assign- 
ments to variables, and contains responses to 
the LRW invocations. We omit details of 
the history formalization, which follow from 
standard techniques similar to the notation 
of transaction serializability. A well-formed 
LRW history is one in which every LRW in- 
vocation finds a matching response. A well- 
formed LRW history determines values for all 
variables. Implementations of LRW or similar 
operations result in refined histories, where 
between invocation and response, lower-level 
events (transmission, reception, message pro- 
cessing) occur. Analysis of such operation his- 
tories, for implementations of operations in 
Table [H can verify their atomicity properties. 

The framework [2j uses terminology of 
transactions to describe local operations, in- 
cluding some ACID properties of transactions 
in databases. Atomicity of a transaction, 
which is the all-or-none property, is due to two 
properties of the protocol. First, in the WSN 
model, ordering transactions can be simple 
because message propagation latency is neg- 
ligible. If nodes p and p' concurrently initi- 
ate a transaction using local broadcast, with 
x,y G N(p) and x,y G N(p'), then x and 
y cannot receive broadcasts from p and p' in 
different order. Second, all the writes of a 
transaction are sandboxed and only actually 
written upon the event of transaction com- 
mit. Consistency of transactions is ensured 
by conflict resolution. If the transactions of 
p and p' conflict, say because they write to 
the same variable in node x, then one of the 
two transactions will be aborted (and possibly 
reinitiated later). Properties corresponding 
to atomicity and consistency can similarly be 
shown for LRW operations (and proved using 
LRW histories). Transactions of p and p' can 
be concurrent, even with neighbors {x, y} in 
common, provided that they operate on dis- 
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tinct sets of variables (and more generally, if it 
can be shown that the transactions have com- 
mutative semantics). This observation also 
holds for LRW operations. 

The problematic aspects of ACID proper- 
ties for WSNs arise from platform limitations 
and unreliable message transport. The pos- 
sibility of message loss implies, for example, 
that a commit message or a cancellation mes- 
sage could be lost. It is well-known that 
no acknowledgment protocol can guarantee 
that all neighbors of a transaction initiator 
will receive a commit or cancellation message, 
even if it is retransmitted some number of 
times [171 US]- However, the probability of 
a communication loss can be reduced if mes- 
sages are retransmitted, and retransmission 
may be a practical strategy to improve reli- 
ability for WSNs (in effect, retransmission is 
an approximation to eventually correct mes- 
sage delivery). A protocol optimization for 
transaction or LRW operations is to replace a 
commit or cancellation message with timeout- 
driven activation. The design choice of [2] and 
in this paper is to let commit be timeout- 
driven: if, after some fixed time period, a 
node does not receive any cancellation mes- 
sage from the LRW initiator, then variables 
writes are committed. An alternative design 
choice would be to let cancellation be the 
timeout-triggered default, however this choice 
would shift the balance of power usage (be- 
cause messages consume power) to commit, 
and for most applications and typical WSN 
workloads, one would expect most LRW op- 
erations to be committed. 

A limitation of several current WSN mes- 
sage protocols is packet payload size. For the 
platform used in our experiments, the pay- 
load is 28 bytes, which limits how much can 
be specified in an LRW operation based on 
a single broadcast. Scaling LRW to larger 
data amounts would require fragmentation of 
LRW message fields over multiple broadcasts. 



Some WSN platforms may not support native 
local broadcast; there, ordering LRW opera- 
tions by the instant of reception would not 
be reliable. However LRW operations can 
also be ordered by timestamp, if the WSN 
has synchronized clocks. With synchronized 
clocks, LRW operations can be grouped by 
slotted time intervals. In a slotted time pro- 
tocol, initiators wait until the beginning of 
a slot before transmitting an LRW operation 
message; when a neighbor receives an LRW 
message, it delays sending a response until 
the end of the current slot, in order to collect 
all LRW operations, order them, and sort out 
conflicts. A reason to consider using unicast, 
rather than broadcast of the initial LRW mes- 
sage, is to improve scheduling efficiency of re- 
sponses from neighbors. The CC2420 radio 
chip has a feature for immediate acknowledg- 
ment of unicast messages, and this feature is 
not available for local broadcast. 

In the discussion above, we have treated 
N(p) as a constant, supposing the neighbor- 
hood of p to be fixed in the WSN. The ex- 
perience of many researchers is that, even for 
a static WSN, radio properties are dynamic: 
the set of stable, bidirectional links defining 
neighborhoods evolves. Therefore the design 
of an LRW protocol should plan for dynamic 
neighborhoods. If an LRW operation fails be- 
cause the initiator did not collect responses 
from every neighbor (this would depend on 
the definition of g), it could be that the neigh- 
borhood has changed. In this case, subse- 
quently submitting the LRW operation would 
use the new neighborhood. 

4 LRW Operation Compari- 
son 

Table Q] does n °t compare expressive, or com- 
puting power of different neighborhood oper- 
ations. In the table, transact consumes most 
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resource, but transact is more powerful than 
any other: in one operation, a function of 
neighborhood values can be computed and 
written to several nodes. An LRW opera- 
tion is strictly less powerful because any value 
written must be prescribed, before the oper- 
ation is invoked, rather than computing the 
value to write during the operation. 

One technique to compare operation power 
is to examine protocols that use only that op- 
eration to solve some classic problem, such 
as consensus. If one operation type enables 
consensus to be solved whereas another op- 
eration does not, then the former operation 
is more powerful (with respect to consensus) 
than the latter. Briefly, a consensus protocol 
begins with each node having an input value 
and a decision variable, which can be written 
at most once. The input is not in any variable, 
that is, input values cannot directly be viewed 
by any of the operations of Table [TJ an early 
step in any consensus protocol is to share the 
input with other nodes. The initial value of 
the decision is some constant lo not equal to 
any node's input. Consensus protocols must 
satisfy three properties: validity, agreement, 
and termination. The termination property is 
that every node eventually writes to its deci- 
sion variable, regardless of the progress or fail- 
ure of other nodes; agreement requires that 
no two nodes write different decision values; 
validity requires that any decision written be 
the input of some node. The difficulty of con- 
sensus lies in the timing of nodes participating 
in the protocol. If some node p is very slow 
to engage in the protocol, then other nodes 
will need to decide without knowing p's in- 
put. Although synchronous timing is implicit 
in the implementation of operations such as 
LRW, software at the application layer may 
be asynchronous, hence the timing of appli- 
cations using LRW can be unpredictable. 

For the following results, we assume com- 
munications are nonlossy and do not fail due 



to contention conflicts. Also, neighborhoods 
are static and definitions of neighborhood are 
consistent, that is, if g G N(p) then p G N(q). 
The following shows that read-all is insuffi- 
cient to solve the consensus problem. 

Lemma 1 Consensus using only read-all op- 
erations is impossible. 

Proof: The proof repeats standard argu- 
ments [1] based on finding a contradiction 
in a constructed execution. Suppose consen- 
sus is possible, and that nodes p and q are 
neighbors with inputs and 1 respectively. If 
p (or q) waits long enough to expose its in- 
put value, then the other node may take suf- 
ficiently many steps so that it is forced, by 
the termination property, to decide; because 
the other's input is unknown, it will decide 
in favor of its own input. Thus the initial 
state for the consensus protocol is multiva- 
lent, that is, there exist two possible execu- 
tions leading to different decisions. A state 
is univalent if all possible executions follow- 
ing that state can only lead to one decision 
(in effect, the decision has already been cho- 
sen, even if not presently in a decision vari- 
able). Executions consist of an interleaving, 
of atomic steps from some node in the neigh- 
borhood, where a step is either a read-all op- 
eration, some local calculation, or writing to 
some variable(s). If p writes to a variable v, 
and the next step in the execution is a read- 
all for v by q, then q obtains the value p wrote 
to v. 

Let a be the last multivalent state in an 
execution (the termination property implies a 
exists). There are at least two possible contin- 
uations from a leading to different decisions, 
by definition. Such continuations necessar- 
ily begin with steps of different nodes. We 
consider different cases for the first step by 
p and q with respect to continuations. Note 
that if p steps first after <r, then the valency 
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is different than would be if q steps first (oth- 
erwise a is not multivalent). If the first step 
by p is a local calculation or a read-all op- 
eration, then the occurrence of that step is 
undetectable by q. This contradicts the as- 
sumption that p's first step after a results in 
a univalent state. If the first step by p writes 
to a variable, then it cannot be that q's first 
step writes to a variable, because these two 
steps commute, which would contradict the 
differing valency of these two steps. There- 
fore, the essential case to examine is where 
p's first step writes to a variable and t/'s first 
step is a read-all operation. If p steps first 
and then sleeps while q runs long enough to 
decide, the valency will follow from p's write 
of a variable; the same valency is obtained if 
q makes no steps while p runs long enough to 
decide. However, p cannot detect whether or 
not q has performed a read-all, hence if q steps 
first, then sleeps, with p running long enough 
to decide, p must decide as if q took no steps, 
which contradicts the supposed valency of q's 
read-all operation. Thus in any case, the tran- 
sition from multivalency to univalency can be 
prevented in some possible execution. □ 

Although read-all doesn't provide a solu- 
tion to consensus, an enhanced form of read- 
all, called read- all- write, does allow for a so- 
lution. In a read-all-write operation, a node 
atomically reads values from all neighbors 
and writes some function of the result to a 
variable. If p and q invoke read-all-write 
at nearly the same instant, then atomicity 
guarantees that one operation will precede 
the other. Thereby, if p's read-all-write oc- 
curs first, then the variable written by p 
will be visible in g's read-all-write. Thus q 
can detect that p's operation preceded q's, 
and the decision value for both nodes can 
be the input of p. A read-all operation is 
"lighter weight" compared to a read-all-write 
operation, which must constrain concurrency 



to guarantee atomicity. We are not aware 
of WSN research on neighborhood read-all- 
write. Presumably the transactional meth- 
ods, say of [7] or [2] could be used to imple- 
ment read-all-write. 

Unlike read-all, the write-all operation can 
be used to solve consensus in particular cases. 
The following first identifies a negative case, 
where write- all is insufficient; afterward we 
discuss a case where a consensus protocol uses 
write-all. 

Lemma 2 Consensus using only single- 
variable write-all operations is impossible. 

Proof: The proof is similar to that for 
Lemma HJ Here, each node obtains values of 
other nodes only by locally reading variables 
that have been assigned by a write-all oper- 
ation. Let a be the last multivalent state in 
an execution, and suppose the next steps of p 
and q are write-all operations. If these steps 
write to different variables, then the steps 
commute and a valency contradiction is ob- 
tained. When N(p) = {q} the steps of p and 
q write to different variables because write-all 
operations assign to variables of other nodes. 
One case where two steps write to the same 
variable, is that p's first step writes to its vari- 
able v p and q 7 s first step is a write-all to v p . 
Suppose the valency of p taking the first step 
is 1, and the valency of q taking the first step 
is 0. If q takes the first step and p sleeps long 
enough for q to decide, the decision is 0. If 
p takes the first step and then sleeps while q 
runs long enough to decide, the decision will 
still be 0, because q overwrote what p had 
written, and thus q is not influenced by p's 
initial step. This contradicts the assumption 
that p's first step results in a univalent state 
with valency 1. □ 

The write-all operation of Table Q] does not 
include any local variable as a write target 
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(that is, p N(p)). An extension to write- 
all would be to include v p in p's write-all of 
variable v. This extension alone turns out not 
to help in solving consensus, however the in- 
clusion of v p together with allowing multiple 
variables to be written does enable a consen- 
sus protocol. Suppose N(p) = {q} and three 
variables u, v, w are initially u. Let p's oper- 
ation write its input value to u and to v; and 
let q's operation write its input value to v and 
to w. Whichever node has the first write-all 
operation forces the decision value to be its 
input. In an execution with differing inputs 
such that p invokes the first write-all and q 
sleeps, p will detect that it has the first op- 
eration, because w = u; and if q does not 
sleep, p may detect that w ^ u>, however then 
v does not contain p's input, and so p detects 
that its write-all occurred first (q will get the 
decision from v in that case). 

It is not difficult to show that LRW or 
transact suffice to solve consensus, because 
it is simple for a node to record a decision 
value that is not overwritten by any subse- 
quent operation. The LRW operation is a 
lighter weight primitive than write-all, which 
deals with more variables than LRW when 
used for for consensus. 

When operations have equivalent solvabil- 
ity power, then may also be compared by the 
time required or the number of operations 
used in a solution. Intuitively an LRW op- 
eration does more work per operation than 
either read-all or write- all operations, and all 
of these have 1-round (optimistic) time com- 
plexity. 

5 Implementation 

The previous sections of the paper motivate 
LRW operations and sketch, at a high level, 
how such an operation could be implemented 
in a WSN. To confirm the feasibility of LRW 



on a current sensor network platform, this 
section reports results from simple experi- 
ments on some small mote networks. Section 
[3] exposes general design issues for an imple- 
mentation, whereas the experimental imple- 
mentation must contend with low-level de- 
sign considerations. For example, the table 
in Figured] reports optimistic message counts 
for LRW, but our experiments consider fail- 
ures in message delivery. Operation duration 
and throughput is affected by thresholds for 
message transit time and expected number of 
retransmissions; such factors are determined 
from experiments. For instance, the duration 
of an LRW operation can be reduced by set- 
ting smaller time limits and lowering the num- 
ber of retries for lost messages; this will allow 
more LRW operations to be executed, at the 
cost of reliability. Figures presented below 
show effects of such tuning decisions, for the 
case of an LRW operation run in isolation and 
also for the case of an LRW contending with 
other operations. 

5.1 Experimental Platform 

Our implementation and experiments were 
written in the NesC language for the TinyOS 
(version 2) operating system, running on 
Telosb [18] and MicaZ motes (both platforms 
use the same radio chip, CC2420). Rather 
than a full implementation of LRW, we used 
a simpler protocol that ignored the case of 
application-triggered operation failures (such 
as one neighbor having a value that cancels 
the LRW operation) ; thus all our experiments 
consider only cases of successful LRW opera- 
tions, except where an operation is rejected 
due to concurrency. The implementation is 
built on several services: a MAC-layer radio 
stack transmits and delivers packets, also in- 
serting random delays (typically between 3ms 
and 12ms) to avoid collision with other trans- 
missions; a neighborhood service determines 
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LRW-initiate(p): 

mode <— active, S <— { } 

Start T Timeout , T Commit 

broadcast ( initMsg p (Tcommit) ) 

Start T Response 

while (mode = active) 
receive(rejectMsg ? ) : 

mode «— cancel 
receive(m = acceptMsg q ) : 
S^SU {sender(m)} 
if |S| = |JV(p)| 

stop Timers; return success 

T Response CXptVeS . 

broadcast (initMsg p (J commit ) ) 

Testavt T Response 

Trimeout expires : 

mode <— abort, S <— { } 
broadcast ( abortMsg p ) 

Start T Response, T Timeout 

while (mode = abort) 

receive(m = abortAck q ) : 

if \S\ = \N(p)\ 

cancel Timers, return canceled 
"^Response expires . 

broadcast ( abortMsg p ) 

Start T Response 

T T imeout expires : 

stop Timers; return failed 



Figure 3: LRW for Initiator p 

the effective set of a node's neighbors (for 
which there is currently bidirectional com- 
munication); a clock synchronization service 
aligns the timers of nodes, which facilitates 
experiments that induce concurrent LRW op- 
erations in a controlled way. Our largest 
experiments used 31 MicaZ motes, and due 
to proximity, we artificially constrained each 
node to have at most six neighbors (essen- 
tially, this is topology control). Our imple- 
mentation employed three countdown timers, 
two for message delivery and acknowledg- 
ment, and a third to limit the total dura- 
tion of the LRW operation. We used two 
timers for message delivery, to make the dis- 
tinction between (i) time for message trans- 
mission and receiving a response message, and 



(ii) the time for successful transmission and 
response from all neighbors, including retries 
of (i). A technical reason to use (ii) instead of 
a retry counter is that the MAC layer's tim- 
ing is randomized, and our design goal was to 
implement a protocol with known thresholds 
for the LRW operation. Should the timer for 
(ii) expire, then the LRW operation's initia- 
tor aborts the operation and transmits abort 
commands to its neighborhood. The third 
timer is for operation commit: if a neighbor 
does not receive an abort message and the 
commit timer expires, then the result of the 
LRW is committed. 

5.2 Three Outcomes for LRW 

We say that an LRW operation has three pos- 
sible outcomes: It is considered a Success if 
the LRW is accepted by all neighbors; the 
operation is considered Canceled if an abort 
message (which may have become necessary 
for a number of reasons) is responded to by 
all neighbors; and it is considered Failed if at 
least one neighbor does not respond to the 
abort message. Each LRW operation returns 
to the application, which invoked LRW, one of 
these three outcomes. The first two outcomes, 
Success and Cancel, are within the intended 
behavior of the protocol (in the terminology 
of transactions, the result satisfies Atomicity 
and Consistency criteria). The final outcome, 
Failed, represents a failure of communication, 
a sensor node crash, or (silent) neighborhood 
reconfiguration. For a failed outcome, it is un- 
certain whether or not all neighbors received 
and committed the LRW operation (possibly, 
the initiator attempted to cancel the opera- 
tion, but not all neighbors acknowledged the 
cancel request within the allowed timeout pe- 
riod). For a set of LRW operations, reliability 
is the percentage of non-failed LRW opera- 
tions, that is, operations which respond by 
success or cancel. 



10 



LRW- neighbor (q) : 

initially: mode — idle 
while (mode = engaged^) 
receive( abortMsg p ) : 

Stop Tcommit 

send( p, abortAckq ) 

mode <— idle 
receive( m — initMsg p (f) ) : 

send( p, acceptMsg^ ) 
receive( initMsg r A r ^ p ) : 

send( r, rejectMsg 9 ) 
Tcommit expires : 

q commits LRW 

mode ^— idle 
while (mode — idle) 

receive( abortMsg,. ) : 

send( r, abortAck, ) 
receive( m — initMsg p (f) ) : 

Start J Commit <— t 

mode +~ engaged p 
send( p, acceptMsg^ ) 



Figure 4: LRW for Neighbor q 
5.3 Protocol 

In the following, p refers to an initiator and 
{(ft, 91, Qk} = N(p) refers to the neigh- 
bors of p. Figures [3] and [5] contain a high- 
level description of the LRW implementation. 
Five message types are used in the proto- 
col, initMsg, acceptMsg, rejectMsg, abortMsg, 
and abortAck. Three event types drive the 
protocol: invocation of an LRW operation, 
message arrival, and timer expiration. The 
protocol timers are denoted J x where x € 
{Response, Timeout, Commit}. Each timer 
starts with some positive value and decreases 
to zero, whereat an expiration event occurs. 
The initiator begins by starting three timers, 
however only JR esponse and T ' Timeout have ex- 
piration events shown in the initiator proto- 
col; T commit provides a time stamp for the 
LRW operation, used by neighbors. (Note 
that T commit could be significant to the ini- 
tiator as a safeguard, to reject any application 



invocation of LRW- initiate while a current in- 
vocation is already in progress; T commit could 
also trigger commit at the initiator itself, but 
to simplify the presentation we suppress such 
details.) 

The first message broadcast by the ini- 
tiator is the initMsg, containing the cur- 
rent Tcommit value and other fields relevant 
to the LRW operation. The initiator waits 
for all neighbors to reply with acceptMsg (if 
^Response expires, then the initiator again 
broadcasts an initMsg). To simplify the de- 
scription, the case where Tcommit expires 
(which could occur if the initiator retries 
a broadcast too many times) is not shown. 
Also, the presentation omits the case where 
a collection of acceptMsg values might trigger 
some application-specific abort of the LRW 
operation. 

For the mote implementation, we also in- 
cluded a list of neighbors in the initMsg: this 
list named the neighbors for which the initia- 
tor had not yet received a acceptMsg. This 
small optimization reduced the overhead af- 
ter retransmitting an initMsg, by avoiding a 
needless resending of acceptMsg (usually for 
the majority of neighbors). 

It is important to note that even if LRW- 
initiate returns Success, Tcommit nmy not 
have expired, and thus the LRW has not been 
committed. For this reason p must not be per- 
mitted to begin a new LRW operation until 
T Commit expires. In our trials, the frequency 
of LRW operations was small enough to en- 
sure that the previous LRW was committed or 
aborted before a new LRW was started, but 
any implementation of this protocol would 
have to address the possibility of such an oc- 
currence. Contrary to this, if the LRW oper- 
ation is aborted, a new LRW may be initiated 
immediately. 

If the LRW fails due to a neighbor qi, then 
since at least one attempt has been made by 
the initiator p to cancel the operation, we 
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know that one of the following is true: (1) q% 
did not receive the initiation message; (2) qi 
received the initiation message, but not the 
abort message; or (3) p did not receive q^s 
response to the abort message. In cases (1) 
and (3) the LRW is not committed, and thus 
consistency is maintained. In Case (2) how- 
ever, the LRW is committed, and consistency 
is lost. 

5.4 Metrics 

We evaluated the LRW implementation with 
respect to time and reliability. For experi- 
ments, we considered different platforms, dif- 
ferent topologies, different low-level choices 
for communication, whether operations are 
run in isolation or operations run are under 
contention, as well as different settings for 
T 'Timeout and J commit- The total time allot- 
ted to an LRW operation is T Commit) how- 
ever under ideal circumstances, all messages 
are sent, delivered, and acknowledged in a 
time possibly much less than J commit- we re- 
fer to the time needed for one LRW opera- 
tion to complete under these circumstances 
as the optimistic duration of an LRW op- 
eration. Measuring the optimistic duration 
is interesting: if the gap between the opti- 
mistic duration and Tcommit is large, and if 
close to ideal circumstances are common, then 
1 Commit may be decreased. A smaller value of 
T commit would allow an application to submit 
more LRW operations, that is, the through- 
put of LRW operations could be higher if the 
operation interval is shorter. However, de- 
creasing Jrimeout and J commit may also de- 
crease the frequency of success, because fewer 
message retries will be attempted and late- 
arriving messages could be discarded. A less 
reliable implementation impacts the applica- 
tion, which may or may not retry an unsuc- 
cessful LRW operation. In view of the two 
unsuccessful outcome possibilities for LRW- 



initiate, canceled and failed, the application 
should decide whether or not to retry a can- 
celed operation. Reducing the frequency of 
failed cases can be handled within the LRW 
implement ation . 
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Figure 5: Broadcast vs Unicast with Ack 

To measure the implementation under con- 
tention, we used the synchronized clock ser- 
vice |19] to arrange that a set of nodes simul- 
taneously invoke LRW-initiate. Within an ex- 
periment, a set of LRW operations started si- 
multaneously is called a series; the duration 
of that series is defined to be the maximum 
optimistic duration of any LRW operation in 
the set. We say that a series failed if at least 
one operation in it failed, and that it succeeded 
otherwise. We define the reliability of a set 
of series as the percentage of successful op- 
erations. Our experiments show factors that 
influence a series duration and reliability. 

5.5 Experiments 

Communication Primitive. Each initMsg 
from an initiator should be acknowledged, ei- 
ther with an acceptMsg or a rejectMsg, by 
each neighbor. The CC2420 radio chip of- 
fers a hardware-level acknowledgment fea- 
ture, which enables the receiver of a unicast 
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message to send an ack frame immediately, 
without the usual MAC delay (to avoid colli- 
sion). Though this feature does not presently 
provide a payload area within the acknowl- 
edgment frame, and hence is not sufficient 
for LRW purposes, we nonetheless tested how 
well using unicast (with the hardware ac- 
knowledgments) compared to using neighbor- 
hood broadcast. Using unicast requires more 
transmission operations by the initiator (one 
per neighbor), but also saves time communi- 
cating acknowledgments. Figure [5] displays 
results of an experiment conducted on Telosb 
motes, where each data point in the graph is 
the mean of approximately 250 successful op- 
erations. Due to a generous timeout, no oper- 
ations were aborted or failed. In this experi- 
ment, there is only one initiator, and the num- 
ber of neighbors varied, shown on the x-axis; 
the optimistic duration is shown on the y-axis. 
Because this experiment showed the superi- 
ority of using broadcast with denser networks 
(and the current unicast with hardware ack is 
deficient for our purposes) , we used the neigh- 
borhood broadcast primitive in all subsequent 
experiments. Figure[5]also suggests a starting 
point for testing different values of ~^ Timeout 
at different neighborhood sizes. For example, 
given an initiator with six neighbors, 50ms 
might be a starting point for an experiment 
testing reliability. 

Reliability and Timeout Recall from 
Section 15.41 that the duration of the T Commit 
timer is a significant factor in the through- 
put of the protocol. If the protocol is ex- 
pected to rapidly execute several LRW oper- 
ations, there is a significant incentive to keep 
the Tcommit timer as short as possible. This 
however carries with it another set of chal- 
lenges: note that the execution of the LRW 
operation must be contained entirely within 
the timespan of T commit- Furthermore, ob- 
serve from Figure [3] that T Timeout performs 



a slightly different function depending on the 
current mode of the initiator p; if it is in ac- 
tive mode the timer is used to enter the ini- 
tiator into abort mode, and if it is in abort 
mode, the timer signals when the operation 
is to be declared as failed. Since the T commit 
timer must be of sufficient duration to allow 
for both of these eventualities it follows that 
the length of T commit must be at least twice 

that of T Timeout- 



Timeout 


Reliability 


50ms 


46.64% 


75ms 


96.87 % 


100ms 


100 % 



Table 2: Reliability without contention. 

Due to the above reasoning, many of our 
experiments were focused on studying the ef- 
fect that reducing the duration of ^Timeout 
would have on the duration and reliability of 
both operations and series. We performed 
two sets of trials. The first set was executed in 
a simple network containing one initiator with 
a static neighborhood of size six. In these ex- 
periments we used TelosB motes. 

80 I 1 

75" 

1 70 T 

| 65 -_ 



55 | 1 =" 

50 75 100 

Timeout in milliseconds 

Figure 6: Average LRW duration with 95% 
confidence interval for varying timeout. 

The second set introduced contention in 
the form of several initiators being present in 
the network at the same time. We also em- 
ployed preexisting clock synchronization and 
neighborhood services. Note however that, 
as was mentioned in Section [3l even for a 
static WSN network the neighborhood of a 
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single mote is often dynamic. The network in 
this case consisted of 31 MicaZ motes, labeled 
as v±,V2, ...,«3i. A mote would be consid- 
ered an initiator if and only if i mod(6) = 1 
(thus the motes v±, v-j, U13, i>i9,f25 and v 3 \ 
were initiators). In order to control the topol- 
ogy of the network, we limited the poten- 
tial neighborhood of an initiator v% such that 

N(Vi) C {Vi-3,Vi-.2,Vi-l,Vi +1 ,Vi + 2,Vi +3 }. 

Observe that because of this, any two coor- 
dinators Vi and v % + 6 will have overlapping 
potential neighborhood (due to using a dy- 
namic neighborhood service, the actual neigh- 
borhood varies). 
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Figure 7: Duration distribution for varying 
timeout. 



T Timeout = 50ms, and incremented this in 
steps of 25ms until we achieved 100% reliabil- 
ity. Table [2] shows the reliability of each trial, 
and Figure [U] shows the average duration of 
the LRW operations with the 95% confidence 
interval. 
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Figure 8: Distribution of operation duration. 



Our initial experiments were intended to 
evaluate the optimistic duration of the pro- 
tocol. The implementation we used for these 
trials did not make use of clock synchroniza- 
tion or neighborhood services. Each trial con- 
sisted of approximately 250 LRW operations 
and the network contained only a single ini- 
tiator with a neighborhood of size six. 

As was previously mentioned, the duration 
of T Timeout is a major factor in the through- 
put. Thus we ran several experiments where 
we varied T Timeout, and in each case noted 
not only the duration of each operation, but 
also the overall reliability. As was suggesting 
in the start of this section, we began with 



As seen in Table [21 with a 50ms timeout 
value the reliability was less than 50%, but 
it increased to approximately 97% when we 
allowed a 75ms timeout, and with a 100ms 
timeout the reliability was 100%. We also see 
from Figure [6] that the average duration of 
the LRW operations increases as the timeout 
is reduced. This may at first seem surpris- 
ing, until we consider the duration distribu- 
tion shown on Figure [71 This figure shows 
the percentage of LRW operations that ended 
within a given time interval, as seen on the x- 
axis. 

There Rl C R few important observations we 
can make when we cross reference Figure [7] 
with Table [21 First, note that an approximate 
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equal percentage of the LRW operations for 
each timeout value ended within the 20-40ms 
interval. Obviously the timeout had no ef- 
fect on these, as one would expect. The next 
large grouping occurs at the 60-80ms time in- 
terval, where we find most of the operations 
from the 100ms timeout trial, and many from 
the 75ms trial. However, the 50ms trial is 
virtually absent from this interval, while it is 
instead almost the sole occupant in the 100- 
120ms interval. Since this interval includes 
the double of the timeout value, it is nat- 
ural to hypothesize that for the majority of 
the LRW operations from the 50ms trial that 
ended within this interval, Trimeout expired 
twice. (To confirm this hypothesis, we in- 
vestigated the behavior of the cycle of broad- 
cast (or rebroadcast triggered by T Response ex- 
piring) and acknowledgments, detailed in the 
next paragraph.) A similar effect is seen for 
the 75ms trial at the 140-160ms interval. The 
only difference is that in this case we note that 
while approximately 10% of the LRW opera- 
tions ended within this interval, the reliability 
data implies only 3% of the operations failed. 
On closer inspection of the data we observed 
that most of the operations that ended within 
this interval were canceled, not failed. In ei- 
ther of the two trials, we see that reducing the 
timeout value had a negative effect not only 
on the reliability, but also on the duration of 
the LRW operations. 
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Figure 9: Average LRW operation duration 
with 95% confidence interval. 



In an experiment using seven TelosB motes, 
each mote acted as initiator approximately 
1,300 times, however the experiment tested 
only the first part of the protocol of Figure E} 
effectively T Timeout = oo for this experiment. 
Each initMsg broadcast included the list of 
neighbors who had not yet responded to the 
LRW operation. The initiator's neighborhood 
was fixed to be the other six motes. Thus, 
the initial broadcast of an initMsg contained 
an invitee list of length six; rebroadcasts con- 
tained smaller invitee lists. An operation ter- 
minated at the instant the initiator had re- 
ceived acknowledgments from all neighbors. 
The experiment allocated approximately four 
seconds to each LRW operation, to ensure 
that no contention between operations could 
occur. In a total of 9,258 operations, 4,228 of 
them (45.6%) were "lucky", that is, acknowl- 
edgments from all six neighbors occurred im- 
mediately following the initial broadcast, so 
that no rebroadcast was necessary. The dis- 
tribution of operation times for these cases is 
shown in Figure [TUl indicated by invited = 6. 
Some 3,288 of the operations included a re- 
broadcast containing a singleton invitee list, 
shown as invited = 1 in the figure. The mean 
(standard deviation) operation duration in 
milliseconds for invited = 6 was 26.5 (3.49), 
while for invited = 1 the results are 19.6 
(3.85). The curves in Figure PTOl are Gaussian 
distributions fitted the the mean and stan- 
dard deviations (we also obtained data for 
invited = k, 1 < k < 6, which follow sim- 
ilar patterns). We also measured the total 
duration (from start to termination) of each 
operation. This is shown in Figure HU clearly 
reflecting ^Response expiration and rebroad- 
cast. The mean number of (re-) broadcasts 
per operation was 1.55, and the bimodal dis- 
tribution in the figure explains this (actually, 
the distribution has three modes, however 
the third does not contribute significantly). 
These experiments with Trimeout = oo ex- 
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plain the results of Figure [7] and Table [2J 
One could even use distributions suggested by 
the fitted curves to build an analytical model, 
however this model would be impractical for 
situations where initiators may contend with 
network traffic and for nonoptimistic cases 
where an LRW operation should be canceled. 
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Our second set of experiments introduced 
contention by having six initiators present in 
the graph. Recall that we enforced a topol- 
ogy on the network such that every initiator 
shares at least one potential neighbor with 
another initiator. In these experiments, each 
trial consisted of approximately 1000 series. 

Due to the dynamic neighborhood, the 
number of motes attached to a single initia- 
tor varied over the course of one trial. Figure 



[9] shows the average LRW operation duration 
with a 95% confidence interval, depending on 
the size of the neighborhood. As we see the 
average duration has increased, especially for 
larger neighborhoods, compared to Figure 
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Figure 12: Distribution of series duration. 



Previously we examined the effect that re- 
ducing the timeout value would have on LRW 
duration and reliability in a non-contention 
network. We repeated these experiments with 
contention, starting with a timeout of 100ms 
and increasing it by 50ms in each trial. Table 
[3] shows the operation and series reliability of 
each trial, and Figure [8] shows the distribution 
of the duration for LRW operations. Figure [8] 
omits the data from the 200 and 300ms trials, 
as these were similar to the 250 and 350ms 
trials. 

As seen in Table [3l the operation reliabil- 
ity is very poor in the 100ms trial. But it 
increases rapidly, becoming approximately 99 
% in the 200ms trial and 100 % at 350ms. 
If we cross reference this with Figure [8] we 
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see that in the 100ms trial almost every LRW 
operation had completed within 220ms, while 
in the 350ms trial every transaction had com- 
pleted within 320ms. 

Due to the inclusion of contention, clock 
synchronization, and dynamic neighbor- 
hoods, the behavior of the protocol is signifi- 
cantly more complex than for previous trials. 
However we still notice the same basic trend 
that was evident in Figure [7J the percent- 
age of operations that ended within 160ms is 
largely similar for each trial, while at later 
intervals we notice first a drop, and then an 
increase in the number of operations from the 
100ms trial. 





Operation 


Series 


Timeout 


Reliability 


Reliability 


100ms 


87.91% 


55.11% 


150ms 


96.65 % 


83.66 % 


200ms 


99.37 % 


96.64 % 


250ms 


99.61 % 


97.85 % 


300ms 


99.85 % 


98.90 % 


350ms 


100.00 % 


100.00 % 



Table 3: Reliability with contention 



This becomes much more pronounced when 
we consider the series reliability in the third 
column in Table [3] and the duration distribu- 
tion in Figure [121 Here we clearly see that a 
large number of the series in the 100ms trial 
ended between 160 and 220ms. 

Recall that we observed from Figure [7J that 
reducing the timeout value has the effect of in- 
creasing the duration of the LRW operations. 
Figure [131 which shows the average series du- 
ration and 95% confidence interval for each 
of the above trials, illustrates the same ten- 
dency. However, note that contrary to what 
we saw in Figure [7J when any reduction of 
the timeout value beyond 100ms resulted in 
an increased duration, we see in this figure 
that the series in the 100ms trial has approx- 
imately the same average duration as in the 



350ms trial. 
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Figure 13: Average series duration with 95% 
confidence interval for varying timeout. 



6 Conclusions 

This paper's proposal and investigation of the 
LRW protocol is an attempt to answer the 
question: what basic, single communication- 
round primitive maximizes the work accom- 
plished for a local neighborhood operation? 
In some sense, an LRW operation is a com- 
munication rendezvous, where members of a 
local neighborhood change states jointly. Not 
surprisingly, such an operation is more pow- 
erful than simpler primitives, such as neigh- 
borhood queries or unconditional broadcast- 
write commands. However, the advantages of 
LRW presume an optimistic, or speculative 
programming approach: if contention is high 
or the semantics of the application do not fa- 
vor success, then LRW could be less attrac- 
tive. 

Without notions of stable storage and jour- 
naling, which support ACID properties of 
database managers, consistency of LRW oper- 
ations cannot be guaranteed; however tuning 
the ^Timeout parameter appropriately does 
increase the probability of non-failed opera- 
tions. The importance of consistency for ab- 
stractions like LRW, write-all, or local trans- 
actions, is diminished for most wireless sensor 
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network applications — lowering component 
cost and conserving power have high priority. 
Applications can often be designed to tolerate 
some small probability of data inconsistency, 
using standard techniques of replication, fil- 
tering, outlier removal, and model-generated 
prediction. 

Our experiments show that tuning 1 'Timeout 
impacts both average duration and reliabil- 
ity. Of the two most significant timers used, 
T Timeout and T Response, we limited ourselves 
to studying the former; J Response was set to 
40 milliseconds in all experiments. A possible 
area of future research would be to examine 
the length of T Response, and its effect on the 
protocol. There are many factors that would 
need to be considered in this case, such as size 
of neighborhood, size of the network, radio ac- 
tivity, and so on. Similar as for J Timeout, we 
expect that there will be trade-offs between 
throughput and the number of unnecessary 
retransmissions. 

One aspect of LRW we did not explore in 
this paper is the "convenience" of LRW for 
common applications of sensor network pro- 
gramming. The introduction explains how 
LRW can be used for local data collection; 
artificial examples of consensus or leader elec- 
tion can easily be shown, however practi- 
cal case studies would be helpful to evaluate 
LRW as a programming primitive. 
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